Is the generalizability of a developed artificial intelligence algorithm for COVID-19 on chest CT sufficient for clinical use? Results from the International Consortium for COVID-19 Imaging AI (ICOVAI).

OBJECTIVES: Only few published artificial intelligence (AI) studies for COVID-19 imaging have been externally validated. Assessing the generalizability of developed models is essential, especially when considering clinical implementation. We report the development of the International Consortium for COVID-19 Imaging AI (ICOVAI) model and perform independent external validation. METHODS: The ICOVAI model was developed using multicenter data (n = 1286 CT scans) to quantify disease extent and assess COVID-19 likelihood using the COVID-19 Reporting and Data System (CO-RADS). A ResUNet model was modified to automatically delineate lung contours and infectious lung opacities on CT scans, after which a random forest predicted the CO-RADS score. After internal testing, the model was externally validated on a multicenter dataset (n = 400) by independent researchers. CO-RADS classification performance was calculated using linearly weighted Cohen's kappa and segmentation performance using Dice Similarity Coefficient (DSC). RESULTS: Regarding internal versus external testing, segmentation performance of lung contours was equally excellent (DSC = 0.97 vs. DSC = 0.97, p = 0.97). Lung opacities segmentation performance was adequate internally (DSC = 0.76), but significantly worse on external validation (DSC = 0.59, p < 0.0001). For CO-RADS classification, agreement with radiologists on the internal set was substantial (kappa = 0.78), but significantly lower on the external set (kappa = 0.62, p < 0.0001). CONCLUSION: In this multicenter study, a model developed for CO-RADS score prediction and quantification of COVID-19 disease extent was found to have a significant reduction in performance on independent external validation versus internal testing. The limited reproducibility of the model restricted its potential for clinical use. The study demonstrates the importance of independent external validation of AI models. KEY POINTS: • The ICOVAI model for prediction of CO-RADS and quantification of disease extent on chest CT of COVID-19 patients was developed using a large sample of multicenter data. • There was substantial performance on internal testing; however, performance was significantly reduced on external validation, performed by independent researchers. The limited generalizability of the model restricts its potential for clinical use. • Results of AI models for COVID-19 imaging on internal tests may not generalize well to external data, demonstrating the importance of independent external validation.

Radiomics-based machine learning differentiates "ground-glass" opacities due to COVID-19 from acute non-COVID-19 lung disease.

Ground-glass opacities (GGOs) are a non-specific high-resolution computed tomography (HRCT) finding tipically observed in early Coronavirus disesase 19 (COVID-19) pneumonia. However, GGOs are also seen in other acute lung diseases, thus making challenging the differential diagnosis. To this aim, we investigated the performance of a radiomics-based machine learning method to discriminate GGOs due to COVID-19 from those due to other acute lung diseases. Two sets of patients were included: a first set of 28 patients (COVID) diagnosed with COVID-19 infection confirmed by real-time polymerase chain reaction (RT-PCR) between March and April 2020 having (a) baseline HRCT at hospital admission and (b) predominant GGOs pattern on HRCT; a second set of 30 patients (nCOVID) showing (a) predominant GGOs pattern on HRCT performed between August 2019 and April 2020 and (b) availability of final diagnosis. Two readers independently segmented GGOs on HRCTs using a semi-automated approach, and radiomics features were extracted using a standard open source software (PyRadiomics). Partial least square (PLS) regression was used as the multivariate machine-learning algorithm. A leave-one-out nested cross-validation was implemented. PLS ß-weights of radiomics features, including the 5% features with the largest ß-weights in magnitude (top 5%), were obtained. The diagnostic performance of the radiomics model was assessed through receiver operating characteristic (ROC) analysis. The Youden's test assessed sensitivity and specificity of the classification. A null hypothesis probability threshold of 5% was chosen (p < 0.05). The predictive model delivered an AUC of 0.868 (Youden's index = 0.68, sensitivity = 93%, specificity 75%, p = 4.2 × 10-7). Of the seven features included in the top 5% features, five were texture-related. A radiomics-based machine learning signature showed the potential to accurately differentiate GGOs due to COVID-19 pneumonia from those due to other acute lung diseases. Most of the discriminant radiomics features were texture-related. This approach may assist clinician to adopt the appropriate management early, while improving the triage of patients.